Skip to content

{lib}[foss/2024a] TensorFlow v2.18.1 w/ CUDA 12.6.0#22921

Merged
boegel merged 10 commits intoeasybuilders:developfrom
pavelToman:20250520193019_new_pr_TensorFlow2181
Oct 9, 2025
Merged

{lib}[foss/2024a] TensorFlow v2.18.1 w/ CUDA 12.6.0#22921
boegel merged 10 commits intoeasybuilders:developfrom
pavelToman:20250520193019_new_pr_TensorFlow2181

Conversation

@pavelToman
Copy link
Collaborator

@pavelToman pavelToman commented May 20, 2025 •
edited
Loading

@github-actions
Copy link

github-actions bot commented May 20, 2025 •
edited
Loading

Updated software TensorFlow-2.18.1-foss-2024a-CUDA-12.6.0.eb

Diff against TensorFlow-2.18.1-foss-2024a.eb

easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.18.1-foss-2024a.eb

diff --git a/easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.18.1-foss-2024a.eb b/easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.18.1-foss-2024a-CUDA-12.6.0.eb
index e840e3624b..60229333a7 100644
--- a/easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.18.1-foss-2024a.eb
+++ b/easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.18.1-foss-2024a-CUDA-12.6.0.eb
@@ -2,6 +2,7 @@ easyblock = 'PythonBundle'
 
 name = 'TensorFlow'
 version = '2.18.1'
+versionsuffix = '-CUDA-%(cudaver)s'
 
 homepage = 'https://www.tensorflow.org/'
 description = "An open-source software library for Machine Intelligence"
@@ -22,6 +23,9 @@ builddependencies = [
 ]
 
 dependencies = [
+    ('CUDA', '12.6.0', '', SYSTEM),
+    ('cuDNN', '9.5.0.50', versionsuffix, SYSTEM),
+    ('NCCL', '2.22.3', versionsuffix),
     ('Python', '3.12.3'),
     ('h5py', '3.12.1'),
     ('cURL', '8.7.1'),
@@ -91,11 +95,16 @@ exts_list = [
             'TensorFlow-2.18.1_fixedpoint.patch',
             'TensorFlow-2.18.1_fix-setup.py.patch',
             'TensorFlow-2.18.1_fix-AVX512-eigen-compilation-gcc13.patch',
+            'TensorFlow-2.18.1_increase-tolerance-Conv3DBackpropFilterV2GradTest.testGradient.patch',
+            'TensorFlow-2.18.1_use-CUDA_HOME-for-libdevice-search.patch',
         ],
         'source_tmpl': 'v%(version)s.tar.gz',
         'source_urls': ['https://github.com/tensorflow/tensorflow/archive/'],
         'test_script': 'TensorFlow-2.x_mnist-test.py',
-        'test_tag_filters_cpu': '-gpu,-tpu,-no_cuda_on_cpu_tap,-no_pip,-no_oss,-oss_serial,-benchmark-test,-v1only',
+        'test_tag_filters_cpu': (
+            '-gpu,-tpu,-no_cuda_on_cpu_tap,'
+            '-no_pip,-no_oss,-oss_serial,-benchmark-test,-v1only'
+        ),
         'test_tag_filters_gpu': (
             'gpu,-no_gpu,-nogpu,-gpu_cupti,-no_cuda11,'
             '-no_pip,-no_oss,-oss_serial,-benchmark-test,-v1only'
@@ -146,6 +155,10 @@ exts_list = [
              '6a30d61fd47b773d5e3a40ba3d02288df16321ffc45500f91b666437d8ec3343'},
             {'TensorFlow-2.18.1_fix-AVX512-eigen-compilation-gcc13.patch':
              '976d4be56144060f2ac4351db8341c6dd72376b0d871d5fddd986548ad03c873'},
+            {'TensorFlow-2.18.1_increase-tolerance-Conv3DBackpropFilterV2GradTest.testGradient.patch':
+             '4f66f4a08e3ad65863fafb2f012ad8343e400df26754aae83030137ae5067f4e'},
+            {'TensorFlow-2.18.1_use-CUDA_HOME-for-libdevice-search.patch':
+             'b71c91478bea357944968d7409c5790404b6a5713e0a270e4e2a78d89c152d77'},
         ],
     }),
 ]
Diff against TensorFlow-2.15.1-foss-2023a-CUDA-12.1.1.eb

easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.15.1-foss-2023a-CUDA-12.1.1.eb

diff --git a/easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.15.1-foss-2023a-CUDA-12.1.1.eb b/easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.18.1-foss-2024a-CUDA-12.6.0.eb
index c9333512ba..60229333a7 100644
--- a/easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.15.1-foss-2023a-CUDA-12.1.1.eb
+++ b/easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.18.1-foss-2024a-CUDA-12.6.0.eb
@@ -1,76 +1,76 @@
 easyblock = 'PythonBundle'
 
 name = 'TensorFlow'
-version = '2.15.1'
+version = '2.18.1'
 versionsuffix = '-CUDA-%(cudaver)s'
 
 homepage = 'https://www.tensorflow.org/'
 description = "An open-source software library for Machine Intelligence"
 
-toolchain = {'name': 'foss', 'version': '2023a'}
+toolchain = {'name': 'foss', 'version': '2024a'}
 toolchainopts = {'pic': True}
 
 builddependencies = [
-    ('Bazel', '6.1.0'),
+    ('hatchling', '1.24.2'),
+    ('Bazel', '6.5.0', '-Java-11'),
     # git 2.x required, see also https://github.com/tensorflow/tensorflow/issues/29053
-    ('git', '2.41.0', '-nodocs'),
-    ('pybind11', '2.11.1'),
+    ('git', '2.45.1'),
+    ('pybind11', '2.12.0'),
     ('UnZip', '6.0'),
     # Required to build some of the extensions
-    ('poetry', '1.5.1'),
-    # Protobuf disabled since 2.13.0 easyconfigs:
-    # Compiling with system protobuf don't seem to work, see:
-    # https://github.com/tensorflow/tensorflow/issues/61593
-    # ('protobuf', '24.0'),
+    ('poetry', '1.8.3'),
+    ('Cython', '3.0.10'),
 ]
 
 dependencies = [
-    ('CUDA', '12.1.1', '', SYSTEM),
-    ('cuDNN', '8.9.2.26', versionsuffix, SYSTEM),
-    ('NCCL', '2.18.3', versionsuffix),
-    ('Python', '3.11.3'),
-    ('h5py', '3.9.0'),
-    ('cURL', '8.0.1'),
-    ('dill', '0.3.7'),
+    ('CUDA', '12.6.0', '', SYSTEM),
+    ('cuDNN', '9.5.0.50', versionsuffix, SYSTEM),
+    ('NCCL', '2.22.3', versionsuffix),
+    ('Python', '3.12.3'),
+    ('h5py', '3.12.1'),
+    ('cURL', '8.7.1'),
+    ('dill', '0.3.9'),
     ('double-conversion', '3.3.0'),
-    ('flatbuffers', '23.5.26'),
-    ('flatbuffers-python', '23.5.26'),
+    ('flatbuffers', '24.3.25'),
+    ('flatbuffers-python', '24.3.25'),
     ('giflib', '5.2.1'),
-    ('hwloc', '2.9.1'),
-    ('ICU', '73.2'),
+    ('hwloc', '2.10.0'),
+    ('ICU', '75.1'),
     ('JsonCpp', '1.9.5'),
-    ('libjpeg-turbo', '2.1.5.1'),
-    ('ml_dtypes', '0.3.2'),
-    ('NASM', '2.16.01'),
-    ('nsync', '1.26.0'),
-    ('SQLite', '3.42.0'),
+    ('libjpeg-turbo', '3.0.1'),
+    ('ml_dtypes', '0.5.0'),
+    ('NASM', '2.16.03'),
+    ('nsync', '1.29.2'),
+    ('SQLite', '3.45.3'),
     ('patchelf', '0.18.0'),
-    ('libpng', '1.6.39'),
-    ('snappy', '1.1.10'),
-    ('zlib', '1.2.13'),
-    ('tensorboard', '2.15.1'),
+    ('libpng', '1.6.43'),
+    ('snappy', '1.2.1'),
+    ('zlib', '1.3.1'),
+    ('grpcio', '1.70.0'),
+    ('wrapt', '1.16.0'),
+    ('Markdown', '3.7'),
+    ('absl-py', '2.1.0'),
+    ('tensorboard', '2.18.0'),
+    ('optree', '0.14.1'),
+    ('typing-extensions', '4.11.0'),
 ]
 
-# Dependencies created and updated using findPythonDeps, see:
-# https://docs.easybuild.io/api/easybuild/scripts/findPythonDeps
-# Notable changes since 2.13.0-foss-2023a
-# - tensoboard-wit deprecated as of tensorboard 2.13.0 (tensorboard@33abcb54d7)
-# - portpicker for tests no longer needed (TF@e85860e838)
-# - opt_einsum now comes from ml_dtypes
 exts_list = [
-    ('wrapt', '1.14.1', {
-        'checksums': ['380a85cf89e0e69b7cfbe2ea9f765f004ff419f34194018a6827ac0e3edfed4d'],
+    ('astor', '0.8.1', {
+        'checksums': ['6a6effda93f4e1ce9f618779b2dd1d9d84f1e32812c23a29b3fff6fd7f63fa5e'],
+    }),
+    ('termcolor', '2.5.0', {
+        'checksums': ['998d8d27da6d48442e8e1f016119076b690d962507531df4890fcd2db2ef8a6f'],
     }),
-    ('termcolor', '2.3.0', {
-        'checksums': ['b5b08f68937f138fe92f6c089b99f1e2da0ae56c52b78bf7075fd95420fd9a5a'],
+    ('Werkzeug', '3.1.3', {
+        'source_tmpl': '%(namelower)s-%(version)s.tar.gz',
+        'checksums': ['60723ce945c19328679790e3282cc758aa4a6040e4bb330f53d30fa546d44746'],
     }),
-    ('tensorflow-estimator', '2.15.0', {
-        'source_tmpl': 'tensorflow_estimator-%(version)s-py2.py3-none-any.whl',
-        'checksums': ['aedf21eec7fb2dc91150fc91a1ce12bc44dbb72278a08b58e79ff87c9e28f153'],
+    ('namex', '0.0.8', {
+        'checksums': ['32a50f6c565c0bb10aa76298c959507abdc0e850efe085dc38f3440fcb3aa90b'],
     }),
-    ('keras', '2.15.0', {
-        'source_tmpl': '%(name)s-%(version)s-py3-none-any.whl',
-        'checksums': ['2dcc6d2e30cf9c951064b63c1f4c404b966c59caf09e01f3549138ec8ee0dd1f'],
+    ('keras', '3.9.2', {
+        'checksums': ['322aab6418ee3de1e2bd0871b60a07f0e444e744a7e8cba79af8b42408879ecf'],
     }),
     ('google-pasta', '0.2.0', {
         'modulename': 'pasta',
@@ -79,29 +79,24 @@ exts_list = [
     ('astunparse', '1.6.3', {
         'checksums': ['5ad93a8456f0d084c3456d059fd9a92cce667963232cbf763eac3bc5b7940872'],
     }),
-    # Required by tests
     ('tblib', '3.0.0', {
         'checksums': ['93622790a0a29e04f0346458face1e144dc4d32f493714c6c3dff82a4adb77e6'],
     }),
-    ('astor', '0.8.1', {
-        'checksums': ['6a6effda93f4e1ce9f618779b2dd1d9d84f1e32812c23a29b3fff6fd7f63fa5e'],
-    }),
     (name, version, {
         'patches': [
-            'TensorFlow-2.1.0_fix-cuda-build.patch',
-            'TensorFlow-2.4.0_dont-use-var-lock.patch',
             'TensorFlow-2.13.0_add-missing-system-protobuf-targets.patch',
-            'TensorFlow-2.13.0_exclude-xnnpack-on-ppc.patch',
-            'TensorFlow-2.15.1_remove-duplicate-gpu-tests.patch',
-            'TensorFlow-2.15.1_remove-libclang-dep.patch',
-            'TensorFlow-2.15.1_remove-io-gcs-filesystem-dep.patch',
-            'TensorFlow-2.15.1_add-default-shell-env.patch',
-            'TensorFlow-2.15.1_fix-AVX512-eigen-compilation.patch',
             'TensorFlow-2.15.1_fix-flatbuffer-license.patch',
             'TensorFlow-2.15.1_fix-pybind11-build.patch',
-            'TensorFlow-2.15.1_fix-cuda_build_defs.patch',
-            'TensorFlow-2.15.1_disable-avx512-extensions.patch',
-            'TensorFlow-2.15.1_disable-tf32-in-fused-matmul-tests.patch'
+            'TensorFlow-2.15.1_add-default-shell-env.patch',
+            'TensorFlow-2.4.0_dont-use-var-lock.patch',
+            'TensorFlow-2.18.1_disable-tf32-in-fused-matmul-tests.patch',
+            'TensorFlow-2.18.1_fix-patchelf.patch',
+            'TensorFlow-2.18.1_fix-xnnpack.patch',
+            'TensorFlow-2.18.1_fixedpoint.patch',
+            'TensorFlow-2.18.1_fix-setup.py.patch',
+            'TensorFlow-2.18.1_fix-AVX512-eigen-compilation-gcc13.patch',
+            'TensorFlow-2.18.1_increase-tolerance-Conv3DBackpropFilterV2GradTest.testGradient.patch',
+            'TensorFlow-2.18.1_use-CUDA_HOME-for-libdevice-search.patch',
         ],
         'source_tmpl': 'v%(version)s.tar.gz',
         'source_urls': ['https://github.com/tensorflow/tensorflow/archive/'],
@@ -132,42 +127,38 @@ exts_list = [
             '-//tensorflow/core/kernels/mkl:mkl_fused_batch_norm_op_test',
             '-//tensorflow/core/ir/importexport/tests/roundtrip/...',
         ],
-        'testopts': '--test_env=HOME=/tmp --test_timeout=3600 --test_size_filters=small ',
+        'testopts': "--test_env=HOME=/tmp --test_timeout=3600 --test_size_filters=small ",
         'testopts_gpu': (
             '--test_env=HOME=/tmp --test_timeout=3600 --test_size_filters=small '
             '--run_under=//tensorflow/tools/ci_build/gpu_build:parallel_gpu_execute '
         ),
         'with_xla': True,
         'checksums': [
-            {'v2.15.1.tar.gz': 'f36416d831f06fe866e149c7cd752da410a11178b01ff5620e9f265511ed57cf'},
-            {'TensorFlow-2.1.0_fix-cuda-build.patch':
-             '78c20aeaa7784b8ceb46238a81e8c2461137d28e0b576deeba8357d23fbe1f5a'},
-            {'TensorFlow-2.4.0_dont-use-var-lock.patch':
-             'b14f2493fd2edf79abd1c4f2dde6c98a3e7d5cb9c25ab9386df874d5f072d6b5'},
+            {'v2.18.1.tar.gz': '467c512b631e72ad5c9d5c16b23669bcf89675de630cfbb58f9dde746d34afa8'},
             {'TensorFlow-2.13.0_add-missing-system-protobuf-targets.patch':
              '77d8c8a5627493fc7c38b4de79d49e60ff6628b05ff969f4cd3ff9857176c459'},
-            {'TensorFlow-2.13.0_exclude-xnnpack-on-ppc.patch':
-             'd0818206846911d946666ded7d3216c0546e37cee1890a2f48dc1a9d71047cad'},
-            {'TensorFlow-2.15.1_remove-duplicate-gpu-tests.patch':
-             'd8810d5b875de5be8603afd743774ce9dd8c0d4a82314c7fe2f284a080be7498'},
-            {'TensorFlow-2.15.1_remove-libclang-dep.patch':
-             '871b2f0221b7a150ac9f563ffad7187e052a7eedd95c20fb4524987d7edb6f21'},
-            {'TensorFlow-2.15.1_remove-io-gcs-filesystem-dep.patch':
-             'eba7351a4b0696c589b9c507bacb0257ebce8c39fde39ab72d5d6a69deaaec02'},
-            {'TensorFlow-2.15.1_add-default-shell-env.patch':
-             '3d5196b4bf2e91048dc8a18f9e8f487a223fcd973d6302e80b0d4000ea3d652b'},
-            {'TensorFlow-2.15.1_fix-AVX512-eigen-compilation.patch':
-             '761059e5f5f5eeeef8aed5517a7685a0eb0a9193d4afe8d45237527681c9c0a3'},
             {'TensorFlow-2.15.1_fix-flatbuffer-license.patch':
              '2c04d5095977a628a238dbf93c5fada7159c86752a7183e64e0cf7c7ab00caf4'},
             {'TensorFlow-2.15.1_fix-pybind11-build.patch':
              '3bb350ac92ab99c63c951c96b3b0160699f5f16822b64f72111ebfd2275cafce'},
-            {'TensorFlow-2.15.1_fix-cuda_build_defs.patch':
-             '091581a7c4fc2fc7af282cab6661632c29029d2f36eccb6695ffa5783e065f88'},
-            {'TensorFlow-2.15.1_disable-avx512-extensions.patch':
-             '506ceecff67237eed9cd9e9e114bc1461f35a343f77f83cb3dab710aa701dc0f'},
-            {'TensorFlow-2.15.1_disable-tf32-in-fused-matmul-tests.patch':
-             'f78aa0e8f814a57e8d2e6b24ff095df49e8654aadb797393fa95a9378d0aa662'},
+            {'TensorFlow-2.15.1_add-default-shell-env.patch':
+             '3d5196b4bf2e91048dc8a18f9e8f487a223fcd973d6302e80b0d4000ea3d652b'},
+            {'TensorFlow-2.4.0_dont-use-var-lock.patch':
+             'b14f2493fd2edf79abd1c4f2dde6c98a3e7d5cb9c25ab9386df874d5f072d6b5'},
+            {'TensorFlow-2.18.1_disable-tf32-in-fused-matmul-tests.patch':
+             '97ec29666f3449f2249b121d3c88a374d888732c978c8f2aa74aa5243c7088cc'},
+            {'TensorFlow-2.18.1_fix-patchelf.patch':
+             '6c9a4d8484868b68aad99527420f22ff09528db588cdb11926ec9340f0c4a816'},
+            {'TensorFlow-2.18.1_fix-xnnpack.patch': 'baf81d7b2b61b5a923cf1f171d5e2400df6b8bd9073d3afa7a356e3bbe11984c'},
+            {'TensorFlow-2.18.1_fixedpoint.patch': '5ea1eb3b32e7df5f9ae711a71778b4cae544b3380f18882186eea035873fb640'},
+            {'TensorFlow-2.18.1_fix-setup.py.patch':
+             '6a30d61fd47b773d5e3a40ba3d02288df16321ffc45500f91b666437d8ec3343'},
+            {'TensorFlow-2.18.1_fix-AVX512-eigen-compilation-gcc13.patch':
+             '976d4be56144060f2ac4351db8341c6dd72376b0d871d5fddd986548ad03c873'},
+            {'TensorFlow-2.18.1_increase-tolerance-Conv3DBackpropFilterV2GradTest.testGradient.patch':
+             '4f66f4a08e3ad65863fafb2f012ad8343e400df26754aae83030137ae5067f4e'},
+            {'TensorFlow-2.18.1_use-CUDA_HOME-for-libdevice-search.patch':
+             'b71c91478bea357944968d7409c5790404b6a5713e0a270e4e2a78d89c152d77'},
         ],
     }),
 ]
Diff against TensorFlow-2.15.1-foss-2023a.eb

easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.15.1-foss-2023a.eb

diff --git a/easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.15.1-foss-2023a.eb b/easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.18.1-foss-2024a-CUDA-12.6.0.eb
index 41a35c0c90..60229333a7 100644
--- a/easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.15.1-foss-2023a.eb
+++ b/easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.18.1-foss-2024a-CUDA-12.6.0.eb
@@ -1,72 +1,76 @@
 easyblock = 'PythonBundle'
 
 name = 'TensorFlow'
-version = '2.15.1'
+version = '2.18.1'
+versionsuffix = '-CUDA-%(cudaver)s'
 
 homepage = 'https://www.tensorflow.org/'
 description = "An open-source software library for Machine Intelligence"
 
-toolchain = {'name': 'foss', 'version': '2023a'}
+toolchain = {'name': 'foss', 'version': '2024a'}
 toolchainopts = {'pic': True}
 
 builddependencies = [
-    ('Bazel', '6.1.0'),
+    ('hatchling', '1.24.2'),
+    ('Bazel', '6.5.0', '-Java-11'),
     # git 2.x required, see also https://github.com/tensorflow/tensorflow/issues/29053
-    ('git', '2.41.0', '-nodocs'),
-    ('pybind11', '2.11.1'),
+    ('git', '2.45.1'),
+    ('pybind11', '2.12.0'),
     ('UnZip', '6.0'),
     # Required to build some of the extensions
-    ('poetry', '1.5.1'),
-    # Protobuf disabled since 2.13.0 easyconfigs:
-    # Compiling with system protobuf don't seem to work, see:
-    # https://github.com/tensorflow/tensorflow/issues/61593
-    # ('protobuf', '24.0'),
+    ('poetry', '1.8.3'),
+    ('Cython', '3.0.10'),
 ]
 
 dependencies = [
-    ('Python', '3.11.3'),
-    ('h5py', '3.9.0'),
-    ('cURL', '8.0.1'),
-    ('dill', '0.3.7'),
+    ('CUDA', '12.6.0', '', SYSTEM),
+    ('cuDNN', '9.5.0.50', versionsuffix, SYSTEM),
+    ('NCCL', '2.22.3', versionsuffix),
+    ('Python', '3.12.3'),
+    ('h5py', '3.12.1'),
+    ('cURL', '8.7.1'),
+    ('dill', '0.3.9'),
     ('double-conversion', '3.3.0'),
-    ('flatbuffers', '23.5.26'),
-    ('flatbuffers-python', '23.5.26'),
+    ('flatbuffers', '24.3.25'),
+    ('flatbuffers-python', '24.3.25'),
     ('giflib', '5.2.1'),
-    ('hwloc', '2.9.1'),
-    ('ICU', '73.2'),
+    ('hwloc', '2.10.0'),
+    ('ICU', '75.1'),
     ('JsonCpp', '1.9.5'),
-    ('libjpeg-turbo', '2.1.5.1'),
-    ('ml_dtypes', '0.3.2'),
-    ('NASM', '2.16.01'),
-    ('nsync', '1.26.0'),
-    ('SQLite', '3.42.0'),
+    ('libjpeg-turbo', '3.0.1'),
+    ('ml_dtypes', '0.5.0'),
+    ('NASM', '2.16.03'),
+    ('nsync', '1.29.2'),
+    ('SQLite', '3.45.3'),
     ('patchelf', '0.18.0'),
-    ('libpng', '1.6.39'),
-    ('snappy', '1.1.10'),
-    ('zlib', '1.2.13'),
-    ('tensorboard', '2.15.1'),
+    ('libpng', '1.6.43'),
+    ('snappy', '1.2.1'),
+    ('zlib', '1.3.1'),
+    ('grpcio', '1.70.0'),
+    ('wrapt', '1.16.0'),
+    ('Markdown', '3.7'),
+    ('absl-py', '2.1.0'),
+    ('tensorboard', '2.18.0'),
+    ('optree', '0.14.1'),
+    ('typing-extensions', '4.11.0'),
 ]
 
-# Dependencies created and updated using findPythonDeps, see:
-# https://docs.easybuild.io/api/easybuild/scripts/findPythonDeps
-# Notable changes since 2.13.0-foss-2023a
-# - tensoboard-wit deprecated as of tensorboard 2.13.0 (tensorboard@33abcb54d7)
-# - portpicker for tests no longer needed (TF@e85860e838)
-# - opt_einsum now comes from ml_dtypes
 exts_list = [
-    ('wrapt', '1.14.1', {
-        'checksums': ['380a85cf89e0e69b7cfbe2ea9f765f004ff419f34194018a6827ac0e3edfed4d'],
+    ('astor', '0.8.1', {
+        'checksums': ['6a6effda93f4e1ce9f618779b2dd1d9d84f1e32812c23a29b3fff6fd7f63fa5e'],
+    }),
+    ('termcolor', '2.5.0', {
+        'checksums': ['998d8d27da6d48442e8e1f016119076b690d962507531df4890fcd2db2ef8a6f'],
     }),
-    ('termcolor', '2.3.0', {
-        'checksums': ['b5b08f68937f138fe92f6c089b99f1e2da0ae56c52b78bf7075fd95420fd9a5a'],
+    ('Werkzeug', '3.1.3', {
+        'source_tmpl': '%(namelower)s-%(version)s.tar.gz',
+        'checksums': ['60723ce945c19328679790e3282cc758aa4a6040e4bb330f53d30fa546d44746'],
     }),
-    ('tensorflow-estimator', '2.15.0', {
-        'source_tmpl': 'tensorflow_estimator-%(version)s-py2.py3-none-any.whl',
-        'checksums': ['aedf21eec7fb2dc91150fc91a1ce12bc44dbb72278a08b58e79ff87c9e28f153'],
+    ('namex', '0.0.8', {
+        'checksums': ['32a50f6c565c0bb10aa76298c959507abdc0e850efe085dc38f3440fcb3aa90b'],
     }),
-    ('keras', '2.15.0', {
-        'source_tmpl': '%(name)s-%(version)s-py3-none-any.whl',
-        'checksums': ['2dcc6d2e30cf9c951064b63c1f4c404b966c59caf09e01f3549138ec8ee0dd1f'],
+    ('keras', '3.9.2', {
+        'checksums': ['322aab6418ee3de1e2bd0871b60a07f0e444e744a7e8cba79af8b42408879ecf'],
     }),
     ('google-pasta', '0.2.0', {
         'modulename': 'pasta',
@@ -75,24 +79,24 @@ exts_list = [
     ('astunparse', '1.6.3', {
         'checksums': ['5ad93a8456f0d084c3456d059fd9a92cce667963232cbf763eac3bc5b7940872'],
     }),
-    # Required by tests
     ('tblib', '3.0.0', {
         'checksums': ['93622790a0a29e04f0346458face1e144dc4d32f493714c6c3dff82a4adb77e6'],
     }),
-    ('astor', '0.8.1', {
-        'checksums': ['6a6effda93f4e1ce9f618779b2dd1d9d84f1e32812c23a29b3fff6fd7f63fa5e'],
-    }),
     (name, version, {
         'patches': [
             'TensorFlow-2.13.0_add-missing-system-protobuf-targets.patch',
-            'TensorFlow-2.13.0_exclude-xnnpack-on-ppc.patch',
-            'TensorFlow-2.15.1_remove-libclang-dep.patch',
-            'TensorFlow-2.15.1_remove-io-gcs-filesystem-dep.patch',
-            'TensorFlow-2.15.1_add-default-shell-env.patch',
-            'TensorFlow-2.15.1_fix-AVX512-eigen-compilation.patch',
             'TensorFlow-2.15.1_fix-flatbuffer-license.patch',
             'TensorFlow-2.15.1_fix-pybind11-build.patch',
-            'TensorFlow-2.15.1_disable-avx512-extensions.patch',
+            'TensorFlow-2.15.1_add-default-shell-env.patch',
+            'TensorFlow-2.4.0_dont-use-var-lock.patch',
+            'TensorFlow-2.18.1_disable-tf32-in-fused-matmul-tests.patch',
+            'TensorFlow-2.18.1_fix-patchelf.patch',
+            'TensorFlow-2.18.1_fix-xnnpack.patch',
+            'TensorFlow-2.18.1_fixedpoint.patch',
+            'TensorFlow-2.18.1_fix-setup.py.patch',
+            'TensorFlow-2.18.1_fix-AVX512-eigen-compilation-gcc13.patch',
+            'TensorFlow-2.18.1_increase-tolerance-Conv3DBackpropFilterV2GradTest.testGradient.patch',
+            'TensorFlow-2.18.1_use-CUDA_HOME-for-libdevice-search.patch',
         ],
         'source_tmpl': 'v%(version)s.tar.gz',
         'source_urls': ['https://github.com/tensorflow/tensorflow/archive/'],
@@ -123,32 +127,38 @@ exts_list = [
             '-//tensorflow/core/kernels/mkl:mkl_fused_batch_norm_op_test',
             '-//tensorflow/core/ir/importexport/tests/roundtrip/...',
         ],
-        'testopts': '--test_env=HOME=/tmp --test_timeout=3600 --test_size_filters=small ',
+        'testopts': "--test_env=HOME=/tmp --test_timeout=3600 --test_size_filters=small ",
         'testopts_gpu': (
             '--test_env=HOME=/tmp --test_timeout=3600 --test_size_filters=small '
             '--run_under=//tensorflow/tools/ci_build/gpu_build:parallel_gpu_execute '
         ),
         'with_xla': True,
         'checksums': [
-            {'v2.15.1.tar.gz': 'f36416d831f06fe866e149c7cd752da410a11178b01ff5620e9f265511ed57cf'},
+            {'v2.18.1.tar.gz': '467c512b631e72ad5c9d5c16b23669bcf89675de630cfbb58f9dde746d34afa8'},
             {'TensorFlow-2.13.0_add-missing-system-protobuf-targets.patch':
              '77d8c8a5627493fc7c38b4de79d49e60ff6628b05ff969f4cd3ff9857176c459'},
-            {'TensorFlow-2.13.0_exclude-xnnpack-on-ppc.patch':
-             'd0818206846911d946666ded7d3216c0546e37cee1890a2f48dc1a9d71047cad'},
-            {'TensorFlow-2.15.1_remove-libclang-dep.patch':
-             '871b2f0221b7a150ac9f563ffad7187e052a7eedd95c20fb4524987d7edb6f21'},
-            {'TensorFlow-2.15.1_remove-io-gcs-filesystem-dep.patch':
-             'eba7351a4b0696c589b9c507bacb0257ebce8c39fde39ab72d5d6a69deaaec02'},
-            {'TensorFlow-2.15.1_add-default-shell-env.patch':
-             '3d5196b4bf2e91048dc8a18f9e8f487a223fcd973d6302e80b0d4000ea3d652b'},
-            {'TensorFlow-2.15.1_fix-AVX512-eigen-compilation.patch':
-             '761059e5f5f5eeeef8aed5517a7685a0eb0a9193d4afe8d45237527681c9c0a3'},
             {'TensorFlow-2.15.1_fix-flatbuffer-license.patch':
              '2c04d5095977a628a238dbf93c5fada7159c86752a7183e64e0cf7c7ab00caf4'},
             {'TensorFlow-2.15.1_fix-pybind11-build.patch':
              '3bb350ac92ab99c63c951c96b3b0160699f5f16822b64f72111ebfd2275cafce'},
-            {'TensorFlow-2.15.1_disable-avx512-extensions.patch':
-             '506ceecff67237eed9cd9e9e114bc1461f35a343f77f83cb3dab710aa701dc0f'},
+            {'TensorFlow-2.15.1_add-default-shell-env.patch':
+             '3d5196b4bf2e91048dc8a18f9e8f487a223fcd973d6302e80b0d4000ea3d652b'},
+            {'TensorFlow-2.4.0_dont-use-var-lock.patch':
+             'b14f2493fd2edf79abd1c4f2dde6c98a3e7d5cb9c25ab9386df874d5f072d6b5'},
+            {'TensorFlow-2.18.1_disable-tf32-in-fused-matmul-tests.patch':
+             '97ec29666f3449f2249b121d3c88a374d888732c978c8f2aa74aa5243c7088cc'},
+            {'TensorFlow-2.18.1_fix-patchelf.patch':
+             '6c9a4d8484868b68aad99527420f22ff09528db588cdb11926ec9340f0c4a816'},
+            {'TensorFlow-2.18.1_fix-xnnpack.patch': 'baf81d7b2b61b5a923cf1f171d5e2400df6b8bd9073d3afa7a356e3bbe11984c'},
+            {'TensorFlow-2.18.1_fixedpoint.patch': '5ea1eb3b32e7df5f9ae711a71778b4cae544b3380f18882186eea035873fb640'},
+            {'TensorFlow-2.18.1_fix-setup.py.patch':
+             '6a30d61fd47b773d5e3a40ba3d02288df16321ffc45500f91b666437d8ec3343'},
+            {'TensorFlow-2.18.1_fix-AVX512-eigen-compilation-gcc13.patch':
+             '976d4be56144060f2ac4351db8341c6dd72376b0d871d5fddd986548ad03c873'},
+            {'TensorFlow-2.18.1_increase-tolerance-Conv3DBackpropFilterV2GradTest.testGradient.patch':
+             '4f66f4a08e3ad65863fafb2f012ad8343e400df26754aae83030137ae5067f4e'},
+            {'TensorFlow-2.18.1_use-CUDA_HOME-for-libdevice-search.patch':
+             'b71c91478bea357944968d7409c5790404b6a5713e0a270e4e2a78d89c152d77'},
         ],
     }),
 ]

@pavelToman
Copy link
Collaborator Author

pavelToman commented May 20, 2025

@boegelbot please test @ jsc-zen3-a100 EB_ARGS="--include-easyblocks-from-pr 3699"

@boegelbot
Copy link
Collaborator

boegelbot commented May 20, 2025

@pavelToman: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=22921 EB_ARGS="--include-easyblocks-from-pr 3699" EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_22921 --ntasks=8 --partition=jsczen3g --gres=gpu:1 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 6407

Test results coming soon (I hope)...

Details

- notification for comment with ID 2895362254 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Collaborator

boegelbot commented May 20, 2025

Test report by @boegelbot
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3699
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
jsczen3g1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.5, x86_64, AMD EPYC-Milan Processor (zen3), 1 x NVIDIA NVIDIA A100 80GB PCIe, 555.42.06, Python 3.9.21
See https://gist.github.com/boegelbot/66ce3bf7419f3dda8481d68a3756bb9d for a full test report.

@pavelToman
Copy link
Collaborator Author

pavelToman commented May 20, 2025

Test report by @boegelbot Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3699 FAILED Build succeeded for 0 out of 1 (1 easyconfigs in total) jsczen3g1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.5, x86_64, AMD EPYC-Milan Processor (zen3), 1 x NVIDIA NVIDIA A100 80GB PCIe, 555.42.06, Python 3.9.21 See https://gist.github.com/boegelbot/66ce3bf7419f3dda8481d68a3756bb9d for a full test report.

Failing without patches from #22848

@pavelToman
Copy link
Collaborator Author

pavelToman commented May 22, 2025

Tests are failing on Joltik with 16 cpus + 36gb ram + eb --parallel 10
https://gist.github.com/pavelToman/13a776984ce61619915dc838d51261e2

@pavelToman
Copy link
Collaborator Author

pavelToman commented May 22, 2025

Tests also failing on accelgor:
https://gist.github.com/pavelToman/1abf157c55da3c87c0d04462f383d635

@pavelToman
Copy link
Collaborator Author

pavelToman commented May 23, 2025 •
edited
Loading

It fails also on Donphan now: https://gist.github.com/pavelToman/1b422fc88ec080014ec4cc0edfd42548
Not on tests but sanity_checks:

2025-05-23 03:12:06.750483: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-05-23 03:12:06.789138: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1747962726.806111 1766159 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1747962726.811646 1766159 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-ampere-ib/software/TensorFlow/2.18.1-foss-2024a-CUDA-12.6.0/lib/python3.12/site-packages/keras/src/layers/reshaping/flatten.py:37: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
I0000 00:00:1747962766.890690 1766159 gpu_device.cc:2022] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13377 MB memory:  -> device: 0, name: NVIDIA A2, pci bus id: 0000:5e:00.0, compute capability: 8.6
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
W0000 00:00:1747962767.560265 1766203 gpu_backend_lib.cc:579] Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice. This may result in compilation or runtime failures, if the program we try to run uses routines from libdevice.
Searched for CUDA in the following directories:
  ./cuda_sdk_lib
  /tmp/vsc47063/easybuild/build/TensorFlow/2.18.1/foss-2024a-CUDA-12.6.0/TensorFlow-2.x_mnist-test.py.runfiles/cuda_nvcc
  /tmp/vs/cuda_nvcc
  /usr/local/cuda
  /scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-ampere-ib/software/TensorFlow/2.18.1-foss-2024a-CUDA-12.6.0/lib/python3.12/site-packages/tensorflow/python/platform/../../../nvidia/cuda_nvcc
  /scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-ampere-ib/software/TensorFlow/2.18.1-foss-2024a-CUDA-12.6.0/lib/python3.12/site-packages/tensorflow/python/platform/../../../../nvidia/cuda_nvcc
  /scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-ampere-ib/software/TensorFlow/2.18.1-foss-2024a-CUDA-12.6.0/lib/python3.12/site-packages/tensorflow/python/platform/../../cuda
  .
You can choose the search directory by setting xla_gpu_cuda_data_dir in HloModule's DebugOptions.  For most apps, setting the environment variable XLA_FLAGS=--xla_gpu_cuda_data_dir=/path/to/cuda will work.
Epoch 1/5
I0000 00:00:1747962770.147719 1766202 service.cc:148] XLA service 0x14a710018a80 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1747962770.147750 1766202 service.cc:156]   StreamExecutor device (0): NVIDIA A2, Compute Capability 8.6
2025-05-23 03:12:50.167527: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
I0000 00:00:1747962770.289885 1766202 cuda_dnn.cc:529] Loaded cuDNN version 90500
2025-05-23 03:12:50.381249: W external/local_xla/xla/service/gpu/autotuning/gemm_fusion_autotuner.cc:868] Compiling 34 configs for 2 fusions on a single thread.
W0000 00:00:1747962773.350501 1766202 gpu_backend_lib.cc:617] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2025-05-23 03:12:53.355638: W tensorflow/core/framework/op_kernel.cc:1841] OP_REQUIRES failed at xla_ops.cc:577 : INTERNAL: libdevice not found at ./libdevice.10.bc
2025-05-23 03:12:53.355714: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: INTERNAL: libdevice not found at ./libdevice.10.bc
	 [[{{node StatefulPartitionedCall}}]]
Traceback (most recent call last):
  File "/tmp/vsc47063/easybuild/build/TensorFlow/2.18.1/foss-2024a-CUDA-12.6.0/TensorFlow-2.x_mnist-test.py", line 20, in <module>
    model.fit(x_train, y_train, epochs=5)
  File "/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-ampere-ib/software/TensorFlow/2.18.1-foss-2024a-CUDA-12.6.0/lib/python3.12/site-packages/keras/src/utils/traceback_utils.py", line 122, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-ampere-ib/software/TensorFlow/2.18.1-foss-2024a-CUDA-12.6.0/lib/python3.12/site-packages/tensorflow/python/eager/execute.py", line 53, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tensorflow.python.framework.errors_impl.InternalError: Graph execution error:
Detected at node StatefulPartitionedCall defined at (most recent call last):
  File "/tmp/vsc47063/easybuild/build/TensorFlow/2.18.1/foss-2024a-CUDA-12.6.0/TensorFlow-2.x_mnist-test.py", line 20, in <module>
  File "/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-ampere-ib/software/TensorFlow/2.18.1-foss-2024a-CUDA-12.6.0/lib/python3.12/site-packages/keras/src/utils/traceback_utils.py", line 117, in error_handler
  File "/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-ampere-ib/software/TensorFlow/2.18.1-foss-2024a-CUDA-12.6.0/lib/python3.12/site-packages/keras/src/backend/tensorflow/trainer.py", line 371, in fit
  File "/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-ampere-ib/software/TensorFlow/2.18.1-foss-2024a-CUDA-12.6.0/lib/python3.12/site-packages/keras/src/backend/tensorflow/trainer.py", line 219, in function
  File "/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-ampere-ib/software/TensorFlow/2.18.1-foss-2024a-CUDA-12.6.0/lib/python3.12/site-packages/keras/src/backend/tensorflow/trainer.py", line 132, in multi_step_on_iterator
libdevice not found at ./libdevice.10.bc
	 [[{{node StatefulPartitionedCall}}]] [Op:__inference_multi_step_on_iterator_1038]

I can see CUDA/12.6.0/nvvm/libdevice/libdevice.10.bc

Should I add XLA_FLAGS with f"--xla_gpu_cuda_data_dir={cuda_root}/nvvm/libdevice to configure_step or something similar to sanity_check_step only?

conda-forge/tensorflow-feedstock#420

But still I do not understand why it worked before on Donphan without XLA_FLAGS set...

@Flamefire
Copy link
Contributor

Flamefire commented May 23, 2025 •
edited
Loading

This is likely caused by TF not using the right flags during build. With some source grepping I found xla_gpu_cuda_data_dir using tsl::CandidateCudaRoots which returns among others a define TF_CUDA_TOOLKIT_PATH which is a template filled by configure with cuda_toolkit_path (third_party/gpus/cuda/cuda_config.h.tpl)

I didn't get much further but I found https://openxla.org/xla/hermetic_cuda?hl=de#pointing_to_cudacudnnnccl_redistributions_on_local_file_system

LOCAL_CUDA_PATH
LOCAL_CUDNN_PATH
LOCAL_NCCL_PATH

--repo_env=LOCAL_CUDA_PATH="/foo/bar/nvidia/cuda" \
--repo_env=LOCAL_CUDNN_PATH="/foo/bar/nvidia/cudnn" \
--repo_env=LOCAL_NCCL_PATH="/foo/bar/nvidia/nccl"

And in the TF sources I found:

    local_cuda_path = get_env_var(repository_ctx, "LOCAL_CUDA_PATH")
    if local_cuda_path:
        _use_local_cuda_path(repository_ctx, local_cuda_path)
    else:
        _use_downloaded_cuda_redistribution(repository_ctx)

git blame returns 2.18.0 as the first release containing it.

Hence this needs an update of the easyblock

Spack also uses: TF_CUDA_PATHS (NCCL, Tensorrt, cuda, cudnn prefixes), not sure if that is required. I'd guess the above are enough

@pavelToman
Copy link
Collaborator Author

pavelToman commented Jun 2, 2025

I add env vars LOCAL_CUDA_PATH, LOCAL_CUDNN_PATH, LOCAL_NCCL_PATH to configure step and either all the --repo_env=LOCAL... to build step, but hitting this error:

 # Configuration: 83a8fcc8d5eacc446943d43c652940a500879683068caf98334bb97736c6f248
 # Execution platform: @local_execution_config_platform//:platform
        /tmp/vsc47063/easybuild/build/TensorFlow/2.18.1/foss-2024a-CUDA-12.6.0/TensorFlow/bazel-root/ff7480e58a9aac53ea1227d23b94aedb/execroot/org_tensorflow/external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc:227: SyntaxWarning: invalid escape sequence '\.'
        re.search('\.cpp$|\.cc$|\.c$|\.cxx$|\.C$', f)]
        external/local_xla/xla/tsl/cuda/cupti_stub.cc:16:10: fatal error: third_party/gpus/cuda/extras/CUPTI/include/cupti.h: No such file or directory
        16 | #include "third_party/gpus/cuda/extras/CUPTI/include/cupti.h"
            |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        compilation terminated.
        Target //tensorflow/tools/pip_package:wheel failed to build

There is CUDA/12.6.0/extras/CUPTI/include/cupti.h

@Flamefire
Copy link
Contributor

Flamefire commented Jun 3, 2025

You might need to ask in the TF repo.

An idea: I found a search for the header at https://github.com/tensorflow/tensorflow/blob/cdced3fe1d6378853f57bcd09e9a8472639a264f/third_party/gpus/find_cuda_config.py#L327

It uses TF_CUDA_PATHS at https://github.com/tensorflow/tensorflow/blob/cdced3fe1d6378853f57bcd09e9a8472639a264f/third_party/gpus/find_cuda_config.py#L596 with the list here suggesting the path ending in CUDA/12.6.0 is required.
But might be a red herring as it seems it would error out if the file is not found.

Interpreting this roughly seems to suggest it will create symlinks.

I assume you only need those repo-env params, not env variables.

@pavelToman
Copy link
Collaborator Author

pavelToman commented Jun 3, 2025

From the log I can see this subcommand:

# Configuration: 36aea02fa9932f6bea72e146b5dab5b9ef7f8fe1e66c66c3aaaf7d1fe9b26fdf
# Execution platform: @local_execution_config_platform//:platform
SUBCOMMAND: # @local_xla//xla/tsl/cuda:cupti_stub [action 'Compiling xla/tsl/cuda/cupti_stub.cc', configuration: 36aea02fa9932f6bea72e146b5dab5b9ef7f8fe1e66c66c3aaaf7d1fe9b26fdf, execution platform: @local_execution_config_platform//:platform]
(cd /tmp/vsc47063/easybuild/build/TensorFlow/2.18.1/foss-2024a-CUDA-12.6.0/TensorFlow/bazel-root/ff7480e58a9aac53ea1227d23b94aedb/execroot/org_tensorflow && \
  exec env - \
    CPATH=/apps/gent/RHEL9/cascadelake-ib/software/cURL/8.7.1-GCCcore-13.3.0/include:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/double-conversion/3.3.0-GCCcore-13.3.0/include:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/flatbuffers/24.3.25-GCCcore-13.3.0/include:/apps/gent/RHEL9/cascadelake-ib/software/giflib/5.2.1-GCCcore-13.3.0/include:/apps/gent/RHEL9/cascadelake-ib/software/hwloc/2.10.0-GCCcore-13.3.0/include:/apps/gent/RHEL9/cascadelake-ib/software/ICU/75.1-GCCcore-13.3.0/include:/apps/gent/RHEL9/cascadelake-ib/software/JsonCpp/1.9.5-GCCcore-13.3.0/include:/apps/gent/RHEL9/cascadelake-ib/software/libjpeg-turbo/3.0.1-GCCcore-13.3.0/include:/apps/gent/RHEL9/cascadelake-ib/software/libpng/1.6.43-GCCcore-13.3.0/include:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/nsync/1.29.2-GCCcore-13.3.0/include:/apps/gent/RHEL9/cascadelake-ib/software/pybind11/2.12.0-GCC-13.3.0/include:/apps/gent/RHEL9/cascadelake-ib/software/snappy/1.2.1-GCCcore-13.3.0/include:/apps/gent/RHEL9/cascadelake-ib/software/SQLite/3.45.3-GCCcore-13.3.0/include:/apps/gent/RHEL9/cascadelake-ib/software/zlib/1.3.1-GCCcore-13.3.0/include:/apps/gent/RHEL9/cascadelake-ib/software/OpenSSL/3/include \
    GCC_HOST_COMPILER_PATH=/tmp/vsc47063/eb-72lyz5gg/tmp0btrlga5/rpath_wrappers/gcc_wrapper/gcc \
    LD_LIBRARY_PATH=/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/protobuf/28.0-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/Abseil/20240722.0-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/snappy/1.2.1-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/libpng/1.6.43-GCCcore-13.3.0/lib:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/nsync/1.29.2-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/libjpeg-turbo/3.0.1-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/JsonCpp/1.9.5-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/giflib/5.2.1-GCCcore-13.3.0/lib:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/flatbuffers/24.3.25-GCCcore-13.3.0/lib:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/double-conversion/3.3.0-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/HDF5/1.14.5-gompi-2024a/lib:/apps/gent/RHEL9/cascadelake-ib/software/Szip/2.1.1-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/SciPy-bundle/2024.05-gfbf-2024a/lib/python3.12/site-packages/numpy/core/lib:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/NCCL/2.22.3-GCCcore-13.3.0-CUDA-12.6.0/lib:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/GDRCopy/2.4.1-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-volta-ib/software/cuDNN/9.5.0.50-CUDA-12.6.0/lib:/apps/gent/RHEL9/cascadelake-volta-ib/software/CUDA/12.6.0/nvvm/lib64:/apps/gent/RHEL9/cascadelake-volta-ib/software/CUDA/12.6.0/extras/CUPTI/lib64:/apps/gent/RHEL9/cascadelake-volta-ib/software/CUDA/12.6.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/ScaLAPACK/2.2.0-gompi-2024a-fb/lib:/apps/gent/RHEL9/cascadelake-ib/software/FFTW.MPI/3.3.10-gompi-2024a/lib:/apps/gent/RHEL9/cascadelake-ib/software/FFTW/3.3.10-GCC-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/FlexiBLAS/3.4.4-GCC-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/OpenBLAS/0.3.27-GCC-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/OpenMPI/5.0.3-GCC-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/UCC/1.3.0-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/PRRTE/3.0.5-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/PMIx/5.0.2-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/UCX/1.16.0-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/libevent/2.1.12-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/hwloc/2.10.0-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/libpciaccess/0.18.1-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/numactl/2.0.18-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/Boost/1.85.0-GCC-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/ICU/75.1-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/zstd/1.5.6-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/lz4/1.9.4-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/gettext/0.22.5-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/libiconv/1.17-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/libxml2/2.12.7-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/expat/2.6.2-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/cURL/8.7.1-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/Java/11.0.20/lib:/apps/gent/RHEL9/cascadelake-ib/software/Python/3.12.3-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/OpenSSL/3/lib:/apps/gent/RHEL9/cascadelake-ib/software/libffi/3.4.5-GCCcore-13.3.0/lib64:/apps/gent/RHEL9/cascadelake-ib/software/XZ/5.4.5-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/SQLite/3.45.3-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/Tcl/8.6.14-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/libreadline/8.2-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/ncurses/6.5-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/bzip2/1.0.8-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/binutils/2.42-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/zlib/1.3.1-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/GCCcore/13.3.0/lib64 \
    LIBRARY_PATH=/apps/gent/RHEL9/cascadelake-ib/software/cURL/8.7.1-GCCcore-13.3.0/lib:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/double-conversion/3.3.0-GCCcore-13.3.0/lib:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/flatbuffers/24.3.25-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/giflib/5.2.1-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/hwloc/2.10.0-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/ICU/75.1-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/JsonCpp/1.9.5-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/libjpeg-turbo/3.0.1-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/libpng/1.6.43-GCCcore-13.3.0/lib:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/nsync/1.29.2-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/pybind11/2.12.0-GCC-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/snappy/1.2.1-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/SQLite/3.45.3-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/zlib/1.3.1-GCCcore-13.3.0/lib:/apps/gent/RHEL9/cascadelake-ib/software/OpenSSL/3/lib \
    PATH=/tmp/vsc47063/eb-72lyz5gg/tmp0btrlga5/rpath_wrappers/ld.bfd_wrapper:/tmp/vsc47063/eb-72lyz5gg/tmp0btrlga5/rpath_wrappers/ld.gold_wrapper:/tmp/vsc47063/eb-72lyz5gg/tmp0btrlga5/rpath_wrappers/ld_wrapper:/tmp/vsc47063/eb-72lyz5gg/tmp0btrlga5/rpath_wrappers/gfortran_wrapper:/tmp/vsc47063/eb-72lyz5gg/tmp0btrlga5/rpath_wrappers/gxx_wrapper:/tmp/vsc47063/eb-72lyz5gg/tmp0btrlga5/rpath_wrappers/gcc_wrapper:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/tensorboard/2.18.0-gfbf-2024a/bin:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/Markdown/3.7-GCCcore-13.3.0/bin:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/protobuf/28.0-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/libpng/1.6.43-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/patchelf/0.18.0-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/libjpeg-turbo/3.0.1-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/NASM/2.16.03-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/giflib/5.2.1-GCCcore-13.3.0/bin:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/flatbuffers/24.3.25-GCCcore-13.3.0/bin:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/dill/0.3.9-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/HDF5/1.14.5-gompi-2024a/bin:/apps/gent/RHEL9/cascadelake-ib/software/SciPy-bundle/2024.05-gfbf-2024a/bin:/apps/gent/RHEL9/cascadelake-ib/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-volta-ib/software/CUDA/12.6.0/nvvm/bin:/apps/gent/RHEL9/cascadelake-volta-ib/software/CUDA/12.6.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/FFTW/3.3.10-GCC-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/FlexiBLAS/3.4.4-GCC-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/OpenMPI/5.0.3-GCC-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/UCC/1.3.0-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/PRRTE/3.0.5-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/PMIx/5.0.2-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/UCX/1.16.0-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/libevent/2.1.12-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/hwloc/2.10.0-GCCcore-13.3.0/sbin:/apps/gent/RHEL9/cascadelake-ib/software/hwloc/2.10.0-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/numactl/2.0.18-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/Cython/3.0.10-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/poetry/1.8.3-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/virtualenv/20.26.2-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/UnZip/6.0-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/pybind11/2.12.0-GCC-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/ICU/75.1-GCCcore-13.3.0/sbin:/apps/gent/RHEL9/cascadelake-ib/software/ICU/75.1-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/zstd/1.5.6-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/lz4/1.9.4-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/gzip/1.13-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/git/2.45.1-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/Perl/5.38.2-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/gettext/0.22.5-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/libiconv/1.17-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/libxml2/2.12.7-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/expat/2.6.2-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/cURL/8.7.1-GCCcore-13.3.0/bin:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/Bazel/6.5.0-GCCcore-13.3.0-Java-11/bin:/apps/gent/RHEL9/cascadelake-ib/software/Java/11.0.20:/apps/gent/RHEL9/cascadelake-ib/software/Java/11.0.20/bin:/apps/gent/RHEL9/cascadelake-ib/software/hatchling/1.24.2-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/Python/3.12.3-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/OpenSSL/3/bin:/apps/gent/RHEL9/cascadelake-ib/software/XZ/5.4.5-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/SQLite/3.45.3-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/Tcl/8.6.14-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/ncurses/6.5-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/bzip2/1.0.8-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/binutils/2.42-GCCcore-13.3.0/bin:/apps/gent/RHEL9/cascadelake-ib/software/GCCcore/13.3.0/bin:/user/gent/470/vsc47063/easybuild/easybuild-framework:/usr/libexec/jobcli:/usr/local/vsc/bin:/opt/VirtualGL/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/user/gent/470/vsc47063/.dotnet/tools \
    PIP_DISABLE_PIP_VERSION_CHECK=true \
    PIP_REQUIRE_VIRTUALENV=false \
    PWD=/proc/self/cwd \
    PYTHONNOUSERSITE=1 \
    PYTHONPATH=/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/TensorFlow/2.18.1-foss-2024a-CUDA-12.6.0/lib/python3.12/site-packages:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/TensorFlow/2.18.1-foss-2024a-CUDA-12.6.0/lib/python3.12/site-packages:/apps/gent/RHEL9/cascadelake-ib/software/typing-extensions/4.11.0-GCCcore-13.3.0/lib/python3.12/site-packages:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/optree/0.14.1-GCCcore-13.3.0/lib/python3.12/site-packages:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/tensorboard/2.18.0-gfbf-2024a/lib/python3.12/site-packages:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/absl-py/2.1.0-GCCcore-13.3.0/lib/python3.12/site-packages:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/Markdown/3.7-GCCcore-13.3.0/lib/python3.12/site-packages:/apps/gent/RHEL9/cascadelake-ib/software/wrapt/1.16.0-gfbf-2024a/lib/python3.12/site-packages:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/grpcio/1.70.0-GCCcore-13.3.0/lib/python3.12/site-packages:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/protobuf-python/5.28.0-GCCcore-13.3.0/lib/python3.12/site-packages:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/ml_dtypes/0.5.0-gfbf-2024a/lib/python3.12/site-packages:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/flatbuffers-python/24.3.25-GCCcore-13.3.0/lib/python3.12/site-packages:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/dill/0.3.9-GCCcore-13.3.0/lib/python3.12/site-packages:/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/h5py/3.12.1-foss-2024a/lib/python3.12/site-packages:/apps/gent/RHEL9/cascadelake-ib/software/mpi4py/4.0.1-gompi-2024a/lib/python3.12/site-packages:/apps/gent/RHEL9/cascadelake-ib/software/SciPy-bundle/2024.05-gfbf-2024a/lib/python3.12/site-packages:/apps/gent/RHEL9/cascadelake-ib/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages:/apps/gent/RHEL9/cascadelake-ib/software/Cython/3.0.10-GCCcore-13.3.0/lib/python3.12/site-packages:/apps/gent/RHEL9/cascadelake-ib/software/poetry/1.8.3-GCCcore-13.3.0/lib/python3.12/site-packages:/apps/gent/RHEL9/cascadelake-ib/software/virtualenv/20.26.2-GCCcore-13.3.0/lib/python3.12/site-packages:/apps/gent/RHEL9/cascadelake-ib/software/cryptography/42.0.8-GCCcore-13.3.0/lib/python3.12/site-packages:/apps/gent/RHEL9/cascadelake-ib/software/cffi/1.16.0-GCCcore-13.3.0/lib/python3.12/site-packages:/apps/gent/RHEL9/cascadelake-ib/software/pybind11/2.12.0-GCC-13.3.0/lib/python3.12/site-packages:/apps/gent/RHEL9/cascadelake-ib/software/hatchling/1.24.2-GCCcore-13.3.0/lib/python3.12/site-packages:/apps/gent/RHEL9/cascadelake-ib/software/Python/3.12.3-GCCcore-13.3.0/easybuild/python \
    PYTHON_BIN_PATH=/apps/gent/RHEL9/cascadelake-ib/software/Python/3.12.3-GCCcore-13.3.0/bin/python \
    PYTHON_LIB_PATH=/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/cascadelake-volta-ib/software/TensorFlow/2.18.1-foss-2024a-CUDA-12.6.0/lib/python3.12/site-packages \
    TF2_BEHAVIOR=1 \
    TF_SYSTEM_LIBS=absl_py,astor_archive,astunparse_archive,boringssl,curl,cython,dill_archive,double_conversion,flatbuffers,functools32_archive,gast_archive,gif,hwloc,icu,jsoncpp_git,libjpeg_turbo,nasm,nsync,org_sqlite,pasta,png,pybind11,six_archive,snappy,tblib_archive,termcolor_archive,typing_extensions_archive,wrapt,zlib \
  external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF bazel-out/k8-opt/bin/external/local_xla/xla/tsl/cuda/_objs/cupti_stub/cupti_stub.pic.d '-frandom-seed=bazel-out/k8-opt/bin/external/local_xla/xla/tsl/cuda/_objs/cupti_stub/cupti_stub.pic.o' '-DEIGEN_MAX_ALIGN_BYTES=64' -DEIGEN_ALLOW_UNALIGNED_SCALARS '-DEIGEN_USE_AVX512_GEMM_KERNELS=0' '-DIMPLIB_EXPORT_SHIMS=1' '-DBAZEL_CURRENT_REPOSITORY="local_xla"' -iquote external/local_xla -iquote bazel-out/k8-opt/bin/external/local_xla -iquote external/local_config_cuda -iquote bazel-out/k8-opt/bin/external/local_config_cuda -iquote external/cuda_cudart -iquote bazel-out/k8-opt/bin/external/cuda_cudart -iquote external/cuda_cublas -iquote bazel-out/k8-opt/bin/external/cuda_cublas -iquote external/cuda_cccl -iquote bazel-out/k8-opt/bin/external/cuda_cccl -iquote external/cuda_nvtx -iquote bazel-out/k8-opt/bin/external/cuda_nvtx -iquote external/cuda_nvcc -iquote bazel-out/k8-opt/bin/external/cuda_nvcc -iquote external/cuda_cusolver -iquote bazel-out/k8-opt/bin/external/cuda_cusolver -iquote external/cuda_cufft -iquote bazel-out/k8-opt/bin/external/cuda_cufft -iquote external/cuda_cusparse -iquote bazel-out/k8-opt/bin/external/cuda_cusparse -iquote external/cuda_curand -iquote bazel-out/k8-opt/bin/external/cuda_curand -iquote external/cuda_cupti -iquote bazel-out/k8-opt/bin/external/cuda_cupti -iquote external/cuda_nvml -iquote bazel-out/k8-opt/bin/external/cuda_nvml -iquote external/cuda_nvjitlink -iquote bazel-out/k8-opt/bin/external/cuda_nvjitlink -iquote external/local_tsl -iquote bazel-out/k8-opt/bin/external/local_tsl -iquote external/eigen_archive -iquote bazel-out/k8-opt/bin/external/eigen_archive -iquote external/ml_dtypes -iquote bazel-out/k8-opt/bin/external/ml_dtypes -iquote external/com_google_absl -iquote bazel-out/k8-opt/bin/external/com_google_absl -iquote external/nsync -iquote bazel-out/k8-opt/bin/external/nsync -iquote external/double_conversion -iquote bazel-out/k8-opt/bin/external/double_conversion -iquote external/com_google_protobuf -iquote bazel-out/k8-opt/bin/external/com_google_protobuf -iquote external/local_config_rocm -iquote bazel-out/k8-opt/bin/external/local_config_rocm -iquote external/local_config_tensorrt -iquote bazel-out/k8-opt/bin/external/local_config_tensorrt -iquote external/nccl_archive -iquote bazel-out/k8-opt/bin/external/nccl_archive -Ibazel-out/k8-opt/bin/external/local_config_cuda/cuda/_virtual_includes/cuda_headers -Ibazel-out/k8-opt/bin/external/cuda_cudart/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_cublas/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_cccl/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_nvtx/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_nvcc/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_cusolver/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_cufft/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_cusparse/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_curand/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_cupti/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_nvml/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_nvjitlink/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/ml_dtypes/_virtual_includes/float8 -Ibazel-out/k8-opt/bin/external/ml_dtypes/_virtual_includes/intn -Ibazel-out/k8-opt/bin/external/local_config_tensorrt/_virtual_includes/tensorrt_headers -Ibazel-out/k8-opt/bin/external/nccl_archive/_virtual_includes/nccl_config -isystem external/local_config_cuda/cuda -isystem bazel-out/k8-opt/bin/external/local_config_cuda/cuda -isystem external/cuda_cudart/include -isystem bazel-out/k8-opt/bin/external/cuda_cudart/include -isystem external/cuda_cublas/include -isystem bazel-out/k8-opt/bin/external/cuda_cublas/include -isystem external/cuda_cccl/include -isystem bazel-out/k8-opt/bin/external/cuda_cccl/include -isystem external/cuda_nvtx/include -isystem bazel-out/k8-opt/bin/external/cuda_nvtx/include -isystem external/cuda_nvcc/include -isystem bazel-out/k8-opt/bin/external/cuda_nvcc/include -isystem external/cuda_cusolver/include -isystem bazel-out/k8-opt/bin/external/cuda_cusolver/include -isystem external/cuda_cufft/include -isystem bazel-out/k8-opt/bin/external/cuda_cufft/include -isystem external/cuda_cusparse/include -isystem bazel-out/k8-opt/bin/external/cuda_cusparse/include -isystem external/cuda_curand/include -isystem bazel-out/k8-opt/bin/external/cuda_curand/include -isystem external/cuda_cupti/include -isystem bazel-out/k8-opt/bin/external/cuda_cupti/include -isystem external/cuda_nvml/include -isystem bazel-out/k8-opt/bin/external/cuda_nvml/include -isystem external/cuda_nvjitlink/include -isystem bazel-out/k8-opt/bin/external/cuda_nvjitlink/include -isystem external/eigen_archive -isystem bazel-out/k8-opt/bin/external/eigen_archive -isystem external/eigen_archive/mkl_include -isystem bazel-out/k8-opt/bin/external/eigen_archive/mkl_include -isystem external/ml_dtypes -isystem bazel-out/k8-opt/bin/external/ml_dtypes -isystem external/ml_dtypes/ml_dtypes -isystem bazel-out/k8-opt/bin/external/ml_dtypes/ml_dtypes -isystem external/com_google_protobuf/src -isystem bazel-out/k8-opt/bin/external/com_google_protobuf/src -isystem external/local_config_rocm/rocm -isystem bazel-out/k8-opt/bin/external/local_config_rocm/rocm -isystem external/local_config_rocm/rocm/rocm/include -isystem bazel-out/k8-opt/bin/external/local_config_rocm/rocm/rocm/include -isystem external/local_config_rocm/rocm/rocm/include/rocrand -isystem bazel-out/k8-opt/bin/external/local_config_rocm/rocm/rocm/include/rocrand -isystem external/local_config_rocm/rocm/rocm/include/roctracer -isystem bazel-out/k8-opt/bin/external/local_config_rocm/rocm/rocm/include/roctracer -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fPIC -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -fno-omit-frame-pointer -no-canonical-prefixes -DNDEBUG -g0 -O2 -ffunction-sections -fdata-sections -Wno-all -Wno-extra -Wno-deprecated -Wno-deprecated-declarations -Wno-ignored-attributes -Wno-array-bounds -Wunused-result '-Werror=unused-result' -Wswitch '-Werror=switch' '-Wno-error=unused-but-set-variable' -DAUTOLOAD_DYNAMIC_KERNELS -O2 -ftree-vectorize '-march=native' -fno-math-errno -fPIC -fPIC '-std=c++17' -c external/local_xla/xla/tsl/cuda/cupti_stub.cc -o bazel-out/k8-opt/bin/external/local_xla/xla/tsl/cuda/_objs/cupti_stub/cupti_stub.pic.o)

The CUPTI should be in CPATH but it is not - maybe this is a problem?

@Flamefire
Copy link
Contributor

Flamefire commented Jun 3, 2025

The error says it can't find "third_party/gpus/cuda/extras/CUPTI/include/cupti.h" and we have it in $EBROOT_CUDA/extras/CUPTI/include/cupti.h, so CPATH is not the issue.

The path is something the build creates. So it either needs a bundled CUDA extracted there (which we don't want) or it should create symlinks at this place which is what I suspect it should do.

As Spack sets TF_CUDA_PATHS I think that is the right one.

Looking at their config file we might need:

  • TF_NEED_TENSORRT = 0/1
  • TF_TENSORRT_VERSION
  • TF_CUDA_VERSION
  • TF_CUDNN_VERSION
  • CUDNN_INSTALL_PATH
  • TF_NCCL_VERSION
  • NCCL_INSTALL_PATH
  • NCCL_HDR_PATH (.../include)
  • TF_CUDA_PATHS (CUDA, CUDNN, TensorRT, NCCL, cuFFT comma-separated paths)
  • CUDA_TOOLKIT_PATH
  • TF_CUDA_COMPUTE_CAPABILITIES
  • HERMETIC_CUDA_COMPUTE_CAPABILITIES

But it seems to not have been updated for a while and checking the configure.py from TF it is:

  • LOCAL_CUDA_PATH
  • LOCAL_CUDNN_PATH
  • LOCAL_NCCL_PATH
  • TF_NEED_CUDA
  • HERMETIC_CUDA_VERSION
  • HERMETIC_CUDNN_VERSION
  • HERMETIC_CUDA_COMPUTE_CAPABILITIES

So I'd set those env vars and pass TF_CUDA_PATHS via --repo-env and if that doesn't work ask in their github

@pavelToman
Copy link
Collaborator Author

pavelToman commented Jun 3, 2025

I can see we have in tensorflow.py: 'TF_CUDA_PATHS': cuda_root for config step. I will try to add it also to build step via --repo-env

@pavelToman
Copy link
Collaborator Author

pavelToman commented Jun 5, 2025

Still the same error with cupti.h, although I add all the vars either to config step and via --repo-env to build step.
@Flamefire could you please try to build it on your nodes?
How is it possible that for @fizwit it works?

@pavelToman
Copy link
Collaborator Author

pavelToman commented Jun 5, 2025

Maybe I found something: jax-ml/jax#23689 (comment)
They have similar problem with XLA in jax and it seems the XLA do not expect local cupti.h in Cuda/extras/CUPTI/include but only in Cuda/include?

@Flamefire
Copy link
Contributor

Flamefire commented Jun 11, 2025

I've had a lively discussion with the XLA devs about that and the result is basically: Don't use a preinstalled CUDA, LOCAL_CUDA_PATH is intended for CUDA devs (NVIDIA), not for end users.

I see 4 options:

  1. Add a patch to TF to recognize the install layout of CUDA (instead of the redist archives)
  2. Copy our CUDA install folder to the build folder using the intended structure
  3. Change our CUDA installation to merge CUPTI with the main folders (we current have extras/CUPTI/lib|include and set paths explicitely to be able to find it)
  4. Don't use our CUDA for the build and let TF download it (we can still specify the major.minor version used)

@Flamefire Flamefire mentioned this pull request Jun 17, 2025
Merged
1 task
@lexming
Copy link
Contributor

lexming commented Jun 19, 2025

If the issue is only caused by the location of a single file like cupti.h, or by a handful of files (e.g. < 20) my vote goes for option 3.

We can symlink those files to their alternatives locations in our installations of CUDA. That will not affect any well-behaved software and should fix the issue with TF.

@Flamefire
Copy link
Contributor

Flamefire commented Jun 20, 2025 

$ find /software/CUDA/12.6.0/extras/CUPTI/include -type f | wc -l
36
$ find /software/CUDA/12.6.0/extras/CUPTI/lib64/ -type f | wc -l
7

Still not too bad though

@lexming lexming mentioned this pull request Jun 20, 2025
Merged
@lexming
Copy link
Contributor

lexming commented Jun 20, 2025 •
edited
Loading

@pavelToman @Flamefire I made a PR for option 3 as discussed: easybuilders/easybuild-easyblocks#3791

@Flamefire
Copy link
Contributor

Flamefire commented Jun 20, 2025

So that might now work with easybuilders/easybuild-easyblocks#3765 I'll test that

@Flamefire
Copy link
Contributor

Flamefire commented Jun 23, 2025

Test report by @Flamefire
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3765
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
n1052 - Linux RHEL 8.9 (Ootpa), x86_64, Intel(R) Xeon(R) Platinum 8470 (sapphirerapids), Python 3.9.18
See https://gist.github.com/Flamefire/5bb8d439a1b7dbd366d3c3400636add6 for a full test report.

@Flamefire
Copy link
Contributor

Flamefire commented Jun 23, 2025

Looks good now.

@pavelToman Can you update this branch to include the missing files please? (rebase on or merge current develop)

@pavelToman
Copy link
Collaborator Author

pavelToman commented Sep 23, 2025 •
edited
Loading

All the files are there. I triggered a rebuild of zlib as well with the last comment, so that we can see if that helps. Some parts on jsc-zen3 are NFS mounted. Maybe this causes some issues (but if so, why haven't we seen this before)?

Thank you! I think the error on JSC was there all the time but before it was hidden by 'excessive module-command related logs', as mention before or by not working gpu from boegelbot tests

@Flamefire
Copy link
Contributor

Flamefire commented Sep 23, 2025

'path_filter': ["/zlib/"] does not work as a fix - it is still failing - I will delete this update

Adding to that: This doesn't influence the relevant part of the compilation.

What is actually the intention of this (besides just trying to workaround the current issue)? Maybe we need another change in the easyblock for that.

@pavelToman
Copy link
Collaborator Author

pavelToman commented Sep 23, 2025 •
edited
Loading

'path_filter': ["/zlib/"] does not work as a fix - it is still failing - I will delete this update

Adding to that: This doesn't influence the relevant part of the compilation.

What is actually the intention of this (besides just trying to workaround the current issue)? Maybe we need another change in the easyblock for that.

This piece of code was added 7 years ago in easybuilders/easybuild-easyblocks#1436 by Damian Alvarez

@Flamefire
Copy link
Contributor

Flamefire commented Sep 23, 2025

Test report by @Flamefire
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
n1295.barnard.hpc.tu-dresden.de - Linux RHEL 8.9 (Ootpa), x86_64, Intel(R) Xeon(R) Platinum 8470 (sapphirerapids), Python 3.9.18
See https://gist.github.com/Flamefire/e7fa128f09b80772b74b753651caa39c for a full test report.

@pavelToman
Copy link
Collaborator Author

pavelToman commented Sep 23, 2025 •
edited
Loading

Test report by @pavelToman
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
node4301.litleo.os - Linux RHEL 9.6, x86_64, AMD EPYC 9454P 48-Core Processor, 1 x NVIDIA NVIDIA H100 NVL, 580.82.07, Python 3.9.21
See https://gist.github.com/pavelToman/93153df0345f3e54b6ba87f78d226218 for a full test report.

Server terminated abruptly (error code: 14, error message: 'Socket closed' -> bazel probably failed to get some sources.

@boegelbot
Copy link
Collaborator

boegelbot commented Sep 23, 2025

Test report by @boegelbot
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
jsczen3g1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.6, x86_64, AMD EPYC-Milan Processor (zen3), 1 x NVIDIA NVIDIA A100 80GB PCIe, 580.82.07, Python 3.9.21
See https://gist.github.com/boegelbot/f3fb269cdaa87100c39ee011d79b7680 for a full test report.

@Flamefire
Copy link
Contributor

Flamefire commented Sep 23, 2025

Test report by @Flamefire
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
n1173.barnard.hpc.tu-dresden.de - Linux RHEL 8.9 (Ootpa), x86_64, Intel(R) Xeon(R) Platinum 8470 (sapphirerapids), Python 3.9.18
See https://gist.github.com/Flamefire/1995575c0b2b34f16c2cc535eeeb45e0 for a full test report.

@pavelToman
Copy link
Collaborator Author

pavelToman commented Sep 23, 2025

Test report by @pavelToman
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
node3902.accelgor.os - Linux RHEL 9.6, x86_64, AMD EPYC 7413 24-Core Processor, 1 x NVIDIA NVIDIA A100-SXM4-80GB, 580.82.07, Python 3.9.21
See https://gist.github.com/pavelToman/3ccbf2eec65e07c590d5051c3bf4df89 for a full test report.

@Flamefire
Copy link
Contributor

Flamefire commented Sep 23, 2025

Test report by @Flamefire
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
i8010 - Linux Rocky Linux 9.6, x86_64, AMD EPYC 7352 24-Core Processor (zen2), 8 x NVIDIA NVIDIA A100-SXM4-40GB, 580.65.06, Python 3.9.21
See https://gist.github.com/Flamefire/b62f68ed543c9e95a926c80b0712e727 for a full test report.

@Flamefire
Copy link
Contributor

Flamefire commented Sep 23, 2025

Test report by @Flamefire
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
i8006 - Linux Rocky Linux 9.6, x86_64, AMD EPYC 7352 24-Core Processor (zen2), 8 x NVIDIA NVIDIA A100-SXM4-40GB, 580.65.06, Python 3.9.21
See https://gist.github.com/Flamefire/c6c3d748b1c04656450c28102d7fa1a2 for a full test report.

@boegel
Copy link
Member

boegel commented Sep 23, 2025

@Thyre Do you have any idea why rebuilding zlib/1.3.1-GCCcore-13.3.0 fixed the problem that was there on jsc-zen3?

@Thyre
Copy link
Collaborator

Thyre commented Sep 23, 2025 •
edited
Loading

@Thyre Do you have any idea why rebuilding zlib/1.3.1-GCCcore-13.3.0 fixed the problem that was there on jsc-zen3?

The module itself looked fine, though I haven‘t checked the contents of all files. Maybe because the module was now in /tmp instead of an NFS-mounted path? But this is just purely guessing.

We can try triggering another build without building in /tmp to see if this makes a difference.

@pavelToman
Copy link
Collaborator Author

pavelToman commented Sep 23, 2025 •
edited
Loading

Test report by @pavelToman
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
node4308.litleo.os - Linux RHEL 9.6, x86_64, AMD EPYC 9454P 48-Core Processor, 1 x NVIDIA NVIDIA H100 NVL, 580.82.07, Python 3.9.21
See https://gist.github.com/pavelToman/ee5522202e0438f5512ab4245efa3f53 for a full test report.

Another weird error on litleo:
ERROR: Could not install packages due to an OSError: [Errno 28] No space left on device: '/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/zen4-hopper-ib/software/TensorFlow/2.18.1-foss-2024a-CUDA-12.6.0/lib/python3.12/site-packages/tensorflow/python/util/_pywrap_transform_graph.so'

@Flamefire
Copy link
Contributor

Flamefire commented Sep 23, 2025 •
edited
Loading

Test report by @Flamefire
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
i8023 - Linux Rocky Linux 9.6, x86_64, AMD EPYC 7352 24-Core Processor (zen2), 8 x NVIDIA NVIDIA A100-SXM4-40GB, 580.65.06, Python 3.9.21
See https://gist.github.com/Flamefire/0a120a60dcad2c82bbd3e27b67b5c84d for a full test report.

Suspected system issue:

C++ exception with description "Bad StatusOr access: INTERNAL: CUDA error: Failed call to cuInit: CUDA_ERROR_SYSTEM_NOT_READY: system not yet initialized" thrown in the test body.

@Flamefire
Copy link
Contributor

Flamefire commented Sep 24, 2025

Test report by @Flamefire
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
c142 - Linux AlmaLinux 9.4, x86_64, AMD EPYC 9334 32-Core Processor (zen4), 4 x NVIDIA NVIDIA H100, 560.35.03, Python 3.9.18
See https://gist.github.com/Flamefire/12c7df3dbd31e66430e3a9874fdaaac5 for a full test report.

@Flamefire
Copy link
Contributor

Flamefire commented Sep 24, 2025

Test report by @Flamefire
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
i8013 - Linux Rocky Linux 9.6, x86_64, AMD EPYC 7352 24-Core Processor (zen2), 8 x NVIDIA NVIDIA A100-SXM4-40GB, 580.65.06, Python 3.9.21
See https://gist.github.com/Flamefire/ea03881a4c14f1e4aee27eabd33194be for a full test report.

@pavelToman
Copy link
Collaborator Author

pavelToman commented Sep 24, 2025 •
edited
Loading

Test report by @pavelToman
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
node4301.litleo.os - Linux RHEL 9.6, x86_64, AMD EPYC 9454P 48-Core Processor, 1 x NVIDIA NVIDIA H100 NVL, 580.82.07, Python 3.9.21
See https://gist.github.com/pavelToman/bfc8e7b72d95b5de1ab1eb9a2e1f1d26 for a full test report.

Again error: ERROR: Could not install packages due to an OSError: [Errno 28] No space left on device: '/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/zen4-hopper-ib/software/TensorFlow/2.18.1-foss-2024a-CUDA-12.6.0/lib/python3.12/site-packages/tensorflow/THIRD_PARTY_NOTICES.txt'

We have a lot of space left on /scratch but we hit a limit for Inodes:

Filesystem     Inodes IUsed IFree IUse% Mounted on
kyukonscratch    6.0M  6.0M     0  100% /kyukon/scratch

@Flamefire
Copy link
Contributor

Flamefire commented Sep 24, 2025

Another weird error on litleo:
ERROR: Could not install packages due to an OSError: [Errno 28] No space left on device: '/scratch/gent/vo/001/gvo00117/easybuild/RHEL9/zen4-hopper-ib/software/TensorFlow/2.18.1-foss-2024a-CUDA-12.6.0/lib/python3.12/site-packages/tensorflow/python/util/_pywrap_transform_graph.so'

TF/Bazel takes an unholy amount of space, multiple GB

@pavelToman
Copy link
Collaborator Author

pavelToman commented Sep 25, 2025

Test report by @pavelToman
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
node4308.litleo.os - Linux RHEL 9.6, x86_64, AMD EPYC 9454P 48-Core Processor, 1 x NVIDIA NVIDIA H100 NVL, 580.82.07, Python 3.9.21
See https://gist.github.com/pavelToman/d60914405772dbb8e812a2b1a938d0d3 for a full test report.

@pavelToman
Copy link
Collaborator Author

pavelToman commented Oct 2, 2025 •
edited
Loading

This seems good to me, what do you think @Flamefire, is it ready to merge?

It success on our gpu nodes litleo(H100) and accelgor(A100), either on jsc-zen3(A100).
I can tests it on our joltik, where is older gpu V100 -> also ok.

@Flamefire
Copy link
Contributor

Flamefire commented Oct 2, 2025

No objections, tests on our machines look good too

@pavelToman
Copy link
Collaborator Author

pavelToman commented Oct 2, 2025

Test report by @pavelToman
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
node3306.joltik.os - Linux RHEL 9.6, x86_64, Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz, 1 x NVIDIA Tesla V100-SXM2-32GB, 580.82.07, Python 3.9.21
See https://gist.github.com/pavelToman/018755dcfa4bc6c25ad20720dacb6ddb for a full test report.

@boegel
Copy link
Member

boegel commented Oct 8, 2025

Test report by @boegel
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
node3906.accelgor.os - Linux RHEL 9.6, x86_64, AMD EPYC 7413 24-Core Processor (zen3), 1 x NVIDIA NVIDIA A100-SXM4-80GB, 580.82.07, Python 3.9.21
See https://gist.github.com/boegel/c0d02fd28798161d691a69e2ae414c88 for a full test report.

@boegel boegel added this to the next release (5.2.0?) milestone Oct 8, 2025
boegel
boegel approved these changes Oct 8, 2025
View reviewed changes
Copy link
Member

@boegel boegel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@boegel
Copy link
Member

boegel commented Oct 8, 2025

Test report by @boegel
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
node3301.joltik.os - Linux RHEL 9.6, x86_64, Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz (cascadelake), 1 x NVIDIA Tesla V100-SXM2-32GB, 580.82.07, Python 3.9.21
See https://gist.github.com/boegel/07b11959f0cd83be08acf2f5db73e97f for a full test report.

@boegel
Copy link
Member

boegel commented Oct 9, 2025

Going in, thanks @pavelToman!

@boegel boegel merged commit 01208fa into easybuilders:develop Oct 9, 2025
8 checks passed
@pavelToman
Copy link
Collaborator Author

pavelToman commented Oct 9, 2025

Thanks go mainly to @Flamefire

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@boegel boegel boegel approved these changes

@lexming lexming lexming left review comments

Assignees

@Flamefire Flamefire

@lexming lexming

Labels

2024a issues & PRs related to 2024a common toolchains ready-to-merge update

Projects

None yet

Milestone

5.2.0

Development

Successfully merging this pull request may close these issues.

7 participants

@pavelToman @boegelbot @Flamefire @lexming @fizwit @Thyre @boegel